Suppose we are interested in classifying a new observation as “John Jay” or “Not John Jay”.
There are many lines we could draw that split the training data perfectly between John Jay and not John Jay!
I’m just drawing any line that separates the two groups. Is there a better way to construct these lines?
How should we choose between these two lines???
The “best” line is the line with the largest margin or is furthest from the nearest observation on either side.
It is rare to have observations that perfectly fall on either side of a line / hyperplane.
Adding one more observation could totally change our classification line!
Suppose we wanted instead to separate “Hamilton” from “Not Hamilton”…
Where should we draw our line?
A soft margin is a margin with only a certain number of misclassified points.
There are two decisions to make here:
(M = width of margin)
(C = cost of a misclassified point)
Width of margin: 2
How many points in the margin are misclassified?
Width of margin: 1
How many points in the margin are misclassified?
The support vector is the set of all observations that falling within the soft margin that are misclassified.
A support vector classifier tries to find:
a line / hyperplane that will be used to classify future observations …
… that give us the biggest margin width (M)…
… while still respecting the cost of missclasified points (C).
tidymodelsInstall a New Package
You will need to install the kernlab package this week!
parsnip model object
Support Vector Machine object of class "ksvm"
SV type: C-svc (classification)
parameter : cost C = 3
Linear (vanilla) kernel function.
Number of Support Vectors : 16
Objective Function Value : -38.0067
Training error : 0.057143
Probability model included.
How well did the model do?
What if we simply couldn’t separate our data with a line / hyperplane?
What if we imagine our points exist in three dimensions?
A support vector machine is an extension of the support vector machine classifer that results from enlarging the feature space in a specific way, using kernels.
In this class, we will only implement polynomial kernals.
Open 07-SVM.qmd to explore how SVM (and PCA) can be used to classify the class of different zoo animals.